Quantum Hall effect of the surface states in topological insulator 
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We study the quantum Hall effect in the surface states of topological insulator in the presence of 
a perpendicular magnetic field in the framework of edge states. Motion of Dirac fermions will form 
descrete Landau levels, among which a fully saturated zero mode will have different behaviors near 
the boundary according to the sign of the effective mass for Dirac fermions. The Hall conductance 
is quantized to be ne 2 /h (n is an integer) for a positive mass, (n + l)e 2 jh for a negative mass, and 
(n + l/2)e 2 /h for massless fermions. In topological insulator the massive term m e // to the Dirac 
fermions can be the Zeeman coupling in a magnetic field or be induced by the finite-size effect in an 
ultrathin film. For example the g- factor of Bi2Se3 is positive and give rise to a positive mass term 
for Dirac fermions. We address experimental realization of the quantum Hall effect in topological 
insulators. 
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Three-dimensional topological insulators possess 
metallic surface states in a band gap which reside near 
the system surface as a consequence of strong spin-orbit 
coupling. 1 Recent experiments and the first principles 
calculations have verified the existence of this novel type 
of topological states in the materials such as Bii^Se^, 
Bi 2 Se 3 , and Bi 2 Te3< 2 -i^£ The surface states have linear 
dispersion in the momenta and are modeled as a two- 
dimensional free Dirac fermion gas, which is analogous 
to a single layer graphene. The Hall conductance of 
two-dimensional Dirac fermions is well defined only 
when the mass m e ff of the fermions is nonvanishing, 

a h = s 9 n ( m ef f) §p 6 I n the presence of a perpendicular 
and uniform magnetic field, it was predicted that 
the Hall conductance of massless Dirac fermions is 
quantized as a half of odd integer (n + 1/2) timing the 
quanta e 2 /h (n is an integer)^ The charge carriers in 
a single layer graphene are regarded as massless Dirac 
fermions, and the measured quantum Hall conductance 
is 4(n + l/2)e 2 /h where the factor 4 originates from 
the spin and valley degeneracy^^ However, strictly 
speaking, the measured quantity in graphene is 2e 2 /h, 
not e 2 /2h directly. The newly discovered topological 
insulator Bi2Se3 and Bi2Te3 have been identified to have 
a single Dirac cone^^ which may provide an ideal 
platform to study a half of the quanta e 2 /h for the Hall 
conductance. On the other hand, the conventional edge 
state picture for quantum Hall effec t 11 ! 12 have estab- 
lished an explicit relation between the Chcrn number 
or the quantized Hall conductance and the number of 
the edge states^ A half-quantized Hall conductance are 
challenging the validity or applicability of the edge state 
theory if the quantum Hall effect exist in the surface 
states of topological insulators. 

In this paper we study the quantum Hall effect for a 
two-dimensional Dirac fermions in the presence of a mag- 
netic field in the edge state theory. The Landau level 
formation will lead to the quantization of the quantum 
Hall conductance. It is found that a fully saturated zero 



mode of the Landau level appears, and its edge effect near 
the boundary determines its contribution to the quantum 
Hall conductance. The sign of the effective mass of Dirac 
fermions plays a decisive role. For Dirac fermions of pos- 
itive mass, the Hall conductance is ne 2 /h as a conven- 
tional two-dimensional electron gas, for Dirac fermions 
of negative mass, the Hall conductance is (n + l)e 2 /h, 
and for massless Dirac fermions the Hall conductance is 
(n + l/2)e 2 /h. These results can be applied to the sur- 
face states of topological insulator and an ultra-thin film 
of topological insulator. Since the surface states in topo- 
logical insulator has a spin texture structure, which is 
distinct from the Dirac cone in graphene, the application 
of magnetic field breaks the time reversal symmetry and 
the Dirac fermions in the surface states acquire a mas- 
sive term, i.e., the Zeeman term. Thus the sign of the 
g-factor will determined the value of quantum Hall con- 
ductance. For example the g-factor in Bi2Te3 is positive 
and generates a positive mass. As a result the quantum 
Hall conductance is ne 2 /h, which is identical as a con- 
ventional two-dimensional electron gas. For an ultra-thin 
film, the finite size confinement will mix the surface states 
at the top and bottom layers and generate an energy gap 
for Dirac fermions. Since the system is still invariant un- 
der time reversal symmetry, one set of fermions possesses 
positive mass and another set possesses negative mass^ 
As a result the Hall conductance can be 2(n + l/2)e 2 /h 
if the energy gap is larger than the Zeeman splitting. 
We start with a 2+1 massive Dirac Hamiltonian 

H = v F h(k x a x + kyO-y) + m e ffV 2 F cr z (1) 

where v e ff is the effective speed of light and m e ff is 
the effective mass. cr x ,y,z are the Pauli matrices. To 
calculate the Hall conductance the Hamiltonian can be 
written in the form, H = d(k) ■ a with the vector d(/c) = 
{vFhk x ,VFhk yi m e f fVp). The Hall conductance of the 
Dirac fermions is only well defined in a massive case, 

e 2 d-(d ka ,dxd ky d) e 2 
ffff = ^? ^jf =s 9 n(m eff) - (2) 



2 



boundaries. In this case, the energy eigenstates are 
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where tan#„. Q = — , = — , a = ±1, n is a positive 

ctWn+S 2 — <5 

integer, |n) = 7^172 ( a ^(?/o))" |0) the solution for a sim- 
ple harmonic oscillator and a(yo) |0) = 0. 17 The Landau 
energy is given by 



E n .a — avF\/2ehB 



(5) 



FIG. 1: (a). Geometry of the sample subjected to a per- 
pendicular magnetic field B. The edge state spectrum of the 
Landau levels (b) for Dirac fermions with negative mass, (c) 
for a massless Dirac fermions, and (d) for Dirac fermions with 
positive mass. 



when the Fermi energy is located between the band 
gap.— i 15 ' 16 For a non-interacting system, it is impossi- 
ble to have a half-quantized Hall conductance in unit of 

e 2 /h. However, the sign change of the mass m e ff will 
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lead a jump of Hall conductance, i.e., A<jh — fr- 

Now we come to study the formation of the Landau lev- 
els with a finite boundary following the theories of edge 
states by Halperinii and MacDonald and Strada^. We 
first consider a geometry of strip with a width L y and 
thickness H, which are much larger than the magnetic 
length Ib and the spatial distribution £ of the surface 
states. Assume the magnetic field B (alone the z-axis) 
is perpendicular to the slab as shown in Fig. 1(a). We 
first focus on the top plane. The periodic boundary con- 
dition is taken along the x-axis, and the open boundary 
condition along the y axis. In this way the wave num- 
ber k x is still a good quantum number, and k y is sub- 
stituted by — id y . We take the Landau gauge for the 
vector potential, A x = +By and A y = 0. Thus we de- 
fine 0(2/0) — ^[®y + 's 2 (y — Vo)] where the magnetic 
length Ib = \JhjeB and yo = —l B k x assuming eB > 0. 
The operators a and satisfy the commutation rela- 
tion, [a (3/0)1 a '(2/0)] = L For simplicity, we introduce a 
dimensionless parameters 5 = m e f fvpls /{^/2vfK). In 
this way, we have a dimensionless Schrodinger equation, 



8 a 

it -6 



■Y2 



E 



^/2eh/B V ^2 



<Pi 



(3) 



The allowed values for yo are separated by Syo = 
2irl B I L x with a periodic boundary condition with length 
L x and are limited within < yo < L y . The solution is a 
function of the good quantum number k x or yo = —l 2 B k x . 
When y is far away from two edges of y = and y = L, 
the two components ip 1 and ip 2 will vanish at the two 



for a positive integer n, which are highly degenerate. 

It should be emphasized that the zero mode Eq = 
—vp\/2ehBS for n = and the eigenstate is fully satu- 
rated, 1 0, 0) = ( V The number of the allowed values 

of yo,^L — Ly/Syo — 2nL x L y /l B , is the degeneracy of 
the Landau levels. The energy expressions yield an en- 
ergy gap AE — |.E n= ±i I — E between the zero mode and 
the states of n — ±1. For 6 — the energy gap is about 
AE w 800K for Bi 2 Se 3 at B = 10T, which makes it pos- 
sible that the quantum Hall effect can be measured even 
at room temperature just as in single layer grapheneJ^ 
The boundary effect will remove the degeneracy of the 
Landau level when yo is near the boundary. It was known 
that we cannot simply take a rigid wall condition such 
that the two components (p± and ip 2 will vanish simulta- 
neously at the boundary. No solution in Eq.© will sat- 
isfy this type of boundary conditions. In the graphene 
it is a common practice to calculate the boundary ef- 
fect numerically in the tight binding approximation.— In 
the present case, since the surface state has a finite spa- 
tial distribution, we do not expect the two components 
ipi and ip 2 "will vanish simultaneously near the boundary. 
We first focus on the boundary effect to the zero mode. 
After some algebra, we have a expression for the zero 
energy mode 



E Vq = — sgn(5)v p \J 2ehB{yo + S 2 



(6) 



where v = (<p 2 \ a f (2/0)0(2/0) |</? 2 ) / (¥2^2) ■ The sign of 
6 guarantees the continuity of the spectrum from the 
bulk to the edge. When << yo « L y , vo = 
and (f 2 {y,y Q ) oc (y\n = 0) = 1 1/2 exp[-(y - yo) 2 /2l%]. 

If the boundary condition makes tp 2 = for yo = 
we have a solution for (p 2 (y,yo — 0) oc (y\n = 1) = 
17 ^3/2 V ex P[~2/ 2 /2^]^ In this case v Q = 1, but in this 

case ip 1 (y 7 yo) oc (y\n = 0). The value of vo varies from 
to 1 when y moves from the bulk to y = or L y . 
In another limit, if we take ip-y = for yo = 0, the solu- 
tion 1 0,0) = [ J and is not distorted by the bound- 
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ary at all, which is apparently unphysical. It will be 
reasonable to assume < vq < 1 near the boundaries, 
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which is between these two limits, since the boundaries 
should distort the Gaussian-type wave function in the 
bulk (0 << yo « L y ). For a negative S, the spectra 
of the Landau levels goes upward near the boundary like 
fcrmions of positive mass, but for a positive 6 the spec- 
trum goes like fermions of negative mass. For massless 
fcrmions of 6 = 0, the energy spectra near the boundary 
have two possible solutions, E UQ — +vpy/2ehBi>o and 
E Va = —vfv^KBvq. Consider the particle-hole symme- 
try. We may say half of the particles in the zero mode 
have positive energy while another half of the particles 
have negative energy. For other states of an integer n, 
v n = (ip 2 1 at(y )a(y ) \<p 2 ) I (Val^) is between n (in the 
bulk) and 2n + l near the edge boundary. In general case, 
we plot the energy spectra schematically in Fig. 1(b), (c) 
and (d) according to the value of 6. 

According to the sign of the mass, m e ff-, these energy 
spectra near the boundary will lead to three different re- 
sults for quantum Hall conductance. For 6 < 0, the zero 
mode has the same behaviors of the states of n > and 
a = +1: the spectrum goes upward near the boundary. 
It will generate an edge current. Following MacDonald 
and Streda^ the edge current is given by 

I„=o = -j — — / dy -l B — -2 G(/i- E Vo {y )) 

oyoLy J h dy 

= fa R -f), (7) 

where ^l r,l are the potentials at the two sides. This will 
contribute one e 2 /h to the Hall conductance. Thus the 
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Hall conductance should be a xy = (n + 1)%- for 5 < 
when other n Landau levels above the zero mode are 
filled. For 8 > 0, the energy spectrum goes downward 
near the boundary. The zero mode will not contribute 
to the Hall conductance when the Fermi energy is above 
the the zero mode, Ef > E Vo , and thus the Hall conduc- 
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tance is titt, which is the quantum Hall conductance for 
a conventional two-dimensional electron gas. The mass- 
less case of 6 = is special since the spectra near the 
boundary can go either upward or down-ward. We may 
say half of the fermions in the zero modes have positive 
mass, and another half have negative mass. In this case 
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the zero mode will contribute one half of the quanta^- to 
the Hall conductance. As a result, the Hall conductance 
should be (n+ 2)%-, which is in agreement with the pre- 
vious results^ In short, the quantum Hall conductance 
for the surface states is determined by the sign of 5 or 
m ef f, 

( (n+l)4 if S < 
*h = \ (n + i)^ if 5 = ■ (8) 
[ n^L. if s > 

Now turn to apply these results to the Dirac fermions 
in the surface states of topological insulators. The Dirac 
fermions are massless and the Dirac point is protected 
by the time reversal symmetry, and is robust against 



impurities or defects. Since the Dirac fermions car- 
ries real spin with a texture structure in the momen- 
tum space, a Zeeman splitting will be induced when the 
system is subjected to a perpendicular magnetic field, 
g e ff/j, B a z , which is equivalent to the effective mass term 
m eff = 9efffJ-B/ v F- Thus the sign of the g- factor will de- 
termine the value of quantum Hall conductance as in Eq. 

For example in Bi2Se3, numerical estimation of the 
value g e f t from the tight binding approximation is about 
g e ff = 1.8^ This illustrates that the Dirac fermions in 
the surface states acquires a positive mass in the presence 
of a perpendicular magnetic field. From this result, we 
conclude that the quantum Hall conductance of the sur- 

2 

face states in Bi 2 Se3 is n^- as in two-dimensional electron 
gas. 

Consider the magnetic field normal to the top and bot- 
tom surfaces of a sample as shown in Fig. 1. The surface 
states of all lateral sides just experience a in-plane field, 
in which the main effect of magnetic field is the Zeeman 
coupling. We have discussed the quantum Hall conduc- 
tance in the top surface. If we assume the magnetic field 
B is along the z-direction, normal to the top surface, the 
electrons in the bottom surface will cquivalcntly experi- 
ence a ~B field. In this case, the operators a and in the 
effective model (Eq.©) for the bottom surface fermions 
should be re-defined: a operator should be replaced by 
and vice versa. The <5-term is attributed to the Zee- 
man term, it will also change its sign, S — > —5. As a re- 
sult, the zero energy mode will change to |0, 0) = ^ 

and the energy remains unchanged, Eq = —vp\/2ehB8. 
Thus we conclude that the Hall conductance a xy of the 
bottom surface is identical to the top surface, and there 
is no voltage drop between the top and bottom surface 
states. Thus the total Hall conductance should be 2ne 2 /h 
(g e ff > 0) or 2(n+ l)e 2 /h (g e ff < 0). These properties 
will make feasible to measure the quantum Hall effect of 
the topological surface states. 

The finite size effect^! of the surface states will gen- 
erate different quantum Hall effect in an ultra-thin film 
of topological insulator. When the spatial distribution 
of the wave function of the surface states is comparable 
with the thickness of the thin film, the quantum tun- 
neling of the wave functions of the top and bottom sur- 
face states will generate an energy gap in the dispersion 
of Dirac fermions i 14 ! 22 ' 23 Since the system is still invari- 
ant under the time reversal symmetry, the two sets of 
Dirac fermions, as a mixture of the top and bottom sur- 
face states, will acquires positive and negative masses, 
m e ffVp = t z A with r = ±1, respectively. The energy 
gap is estimated to be about 0.13eV when the thickness 
of thin film is 20A. The effective mass term for Dirac 
fermions in an ultra-thin film becomes 

AH=(T z A + g efft i B B)a z . (9) 

The value of the energy gap A is dependent of the film 
thickness and decays exponentially to zero. If the gap is 
smaller than the Zeeman splitting, the Hall conductance 
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will be determined by the sign of the g-factor. However, 
if the gap is larger than the Zeeman splitting, the effec- 
tive mass is positive for one set of fermions and negative 
for another set. In this case the quantum Hall conduc- 
tance for the two sets of Dirac fermions are (n + 1)4- 

2 

and Tij-, respectively. As a result the total quantum 

Hall conductance is 2(n+ which looks like the Hall 
conductance for doubly degenerated and massless Dirac 
fermions. Lee^l obtained the quantized Hall conductance 
2(rt + i)^- in the same geometry, which is consistent with 
the present result by neglecting the Zeeman splitting. 

In conclusion, the Zeeman coupling will be induced as 
a mass term for Dirac fermions in the topological surface 
states in the presence of a magnetic field. From the point 
of view of edge states, the sign of the effective g-factor 
or mass determines the dispersion behaviors of Landau 



levels near the boundary, and further the value of the 
quantum Hall conductance of the Dirac fermions. For 
Bi 2 Se3 the calculated g-factor is positive, or equivalently 
the Dirac fermions have positive mass, which gives rise 
to the quantum Hall conductance 2ne 2 //i,as the conven- 
tional two-dimensional electron gas. For an ultra-thin 
film, the finite-size effect will open an energy gap for 
Dirac fermions. If the gap is smaller than the Zeeman 
splitting, the Hall conductance is 2ne 2 /h while if it is 
larger than Zeeman splitting, the quantum Hall conduc- 
tance is 2(n + \)\r- 
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